<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
    <head>
        <title>RVM : Test Run Descriptions</title>
	    <link rel="stylesheet" href="styles/site.css" type="text/css" />
        <META http-equiv="Content-Type" content="text/html; charset=UTF-8">	    
    </head>

    <body>
	    <table class="pagecontent" border="0" cellpadding="0" cellspacing="0" width="100%" bgcolor="#ffffff">
		    <tr>
			    <td valign="top" class="pagebody">
				    <div class="pageheader">
					    <span class="pagetitle">
                            RVM : Test Run Descriptions
                                                    </span>
				    </div>
				    <div class="pagesubheading">
					    This page last changed on Jun 13, 2007 by <font color="#0050B2">pdonald</font>.
				    </div>

				    <p>The Jikes RVM project contains several different test runs with different purposes. This document attempts to capture the purpose of each different test run.</p>

<h2><a name="TestRunDescriptions-"></a><font color="#ff0033">Red test run</font></h2>

<p>This test run <b>MUST</b> be run prior to committing code. They are relatively short and are designed to capture as many potential bugs in the shortest possible time.It is expected that the red test run will take 15-20 minutes on modern intel architecture.</p>

<h2><a name="TestRunDescriptions-"></a><font color="#66cc00">Green test run</font></h2>

<p>There is a set of workloads we consider important (i.e. dacapo and SPEC*). There is a set of build configurations we consider important (ie prototype, development, production). We as a group wish to guarantee that all important workloads will will run correctly on all important build configurations. ( i.e. We should <b>NEVER</b> regress). The green test run is designed to identify as early as possible any failures in this matrix of build configuration x workload. It is run continuously 24 hours a day (or at least every time a change is made). It is expected that the green test run will take 2-6 hours to complete depending on the environment.</p>

<p>The best way to identify the failures is to stress test the system by forcing frequent garbage collections and compilation at specific optimization levels (and perhaps frequent thread switching and frequent OSR events in the future). It is critical that we have a stable research base so intermittent failures are NOT acceptable. If we can not pass a stress test then there is no guarantee that we have a stable research base.</p>

<h2><a name="TestRunDescriptions-"></a><font color="#3300ff">Blue test run</font></h2>

<p>The blue test run cover a larger number of build configurations and workloads. They may not always pass and may test many of the less frequently used configurations (gctrace, gcspy, and individual stress tests) and less important workloads. Performance tests are also included in this test run. Something we use to gauge the health of the project as a whole and to track regressions. These are run once a day on major platforms. These time to complete can vary but expected to take several hours at the least.</p>

<h2><a name="TestRunDescriptions-"></a><font color="#9900cc">Ra</font><font color="#0000ff">inb</font><font color="#009933">ow</font> <font color="#ff9900">te</font><font color="#ff66cc">st r</font><font color="#ff0033">uns</font></h2>

<p>This is not a single test run but a set of test runs that are used for testing specific aspects of the system from performance, gcmap bug finding,  io hammering etc. There may also be a set of personal/site-specific test runs included in this set that are not checked into subversion repository.</p>

<h2><a name="TestRunDescriptions-Summary"></a>Summary</h2>

<p>We must <b>NEVER</b> regress in green test run. The red test run attempts to ensure no green regressions this while keeping running time reasonable. The blue test run gives us an overall picture on the health of the code base. While the rainbow test runs are used at different times for different purposes. </p>

				    
                    			    </td>
		    </tr>
	    </table>
	    <table border="0" cellpadding="0" cellspacing="0" width="100%">
			<tr>
				<td height="12" background="http://docs.codehaus.org/images/border/border_bottom.gif"><img src="images/border/spacer.gif" width="1" height="1" border="0"/></td>
			</tr>
		    <tr>
			    <td align="center"><font color="grey">Document generated by Confluence on Jun 10, 2009 09:12</font></td>
		    </tr>
	    </table>
    </body>
</html>